Automatic terrain evaluation of landing surfaces, and associated systems and methods

ABSTRACT

Automatic terrain evaluation of landing surfaces, and associated systems and methods are disclosed herein. A representative method includes receiving a request to land a movable object and, in response to the request, identifying a target landing area on a landing surface based on at least one image of the landing surface obtained by the movable object. The method can further include directing the movable object to land at the target landing area.

TECHNICAL FIELD

The present technology is directed generally to devices thatautomatically measure or otherwise evaluate the terrain of a landingsurface or other target surface, and associated systems and methods.

BACKGROUND

Unmanned aerial vehicles (UAVs) are aircraft that operate without humanpilots aboard. UAVs may operate under remote control via off-board humancontrollers, or autonomously by onboard computers. Use of UAVs israpidly expanding in military, commercial, scientific, recreational andother applications.

SUMMARY

The following summary is provided for the convenience of the reader andidentifies several representative embodiments of the disclosedtechnology.

A computer-implemented method for automatically identifyingcharacteristics of a landing surface in accordance with a representativeembodiment includes receiving a request to land a movable object; inresponse to the request, identifying a target landing area on a landingsurface based on at least one image of the landing surface obtained bythe movable object; and directing the movable object to land at thetarget landing area. In particular embodiments, the method can includeretrieving depth information corresponding to individual pixels of theat least one image of the landing surface, wherein the individual pixelscorrespond to portions of the landing surface. In any of the foregoingembodiments, the method can include generating a depth map including thedepth information of the individual pixels, which can represent theheights of the corresponding portions of the landing surface. In any ofthe foregoing embodiments, the at least one image can include at leasttwo images, and the method can include performing a stereo matching onthe at least two images of the landing surface to generate depthinformation of matching pixels of the at least two images.

In any of the foregoing embodiments, the method can include identifyinga plurality of candidate areas within a pixel frame of the at least oneimage of the landing surface. The method can further include, for anindividual candidate area, identifying a best fit plane for pixelswithin the candidate area and determining a cost value associated withthe best fit plane based on a cost function. The method can stillfurther include selecting at least one of the candidate areas ascorresponding to the target landing area, based on the cost values. Inany of the foregoing embodiments, the cost value associated with thebest fit plane can denote a degree of deviation of pixels from the bestfit plane. In any of the foregoing embodiments, the cost function can bebased on a sum of squares of deviations of pixels from the best fitplane. In any of the foregoing embodiments, the method can includedetermining whether the best fit planes are tilted by comparing normalvectors of the best fit planes with a gravity vector, and discardingcandidate areas with best fit planes that are tilted. In any of theforegoing embodiments, the method can include discarding candidate areaswith cost functions that are larger than a threshold value. In any ofthe foregoing embodiments, the method can include (if all candidateareas are discarded because the cost functions are larger than thethreshold value) directing the movable object to travel to anotherlocation, and capturing images of another landing surface at the otherlocation.

In any of the foregoing embodiments, the method can include adjustingthe cost functions to include cost function information for nearbycandidate areas. In any of the foregoing embodiments, the process ofadjusting the cost function can include applying a smoothing function toadjust the cost functions of the individual candidate areas. In any ofthe foregoing embodiments, the smoothing function can be a Gaussian blurfunction. In any of the foregoing embodiments, the process of selectingcan include selecting the candidate area with the lowest cost functionof the individual candidate areas. In any of the foregoing embodiments,the process of determining a plurality of candidate areas can includescanning the pixel frame with a sliding window to generate the pluralityof candidate areas, wherein a size of the sliding window is equal to orlarger than a size of the movable object. In any of the foregoingembodiments, a shape of an individual candidate area can conform to ashape of the movable object.

In other embodiments, a representative apparatus includes at least onemotor, and a controller programmed with instructions. The instructions,when executed, receive a request to land the apparatus, and, in responseto the request, identify a target landing area on a landing surfacebased on at least one image of the landing surface obtained by theapparatus. The instructions can further direct the apparatus to land atthe target landing area using the at least one motor. In particularembodiments, the apparatus can be an unmanned aerial vehicle (UAV), amanned aircraft, an autonomous car, a self-balancing vehicle, or arobot. In any of the foregoing embodiments, the instructions, whenexecuted, can capture new images of the landing surface as the apparatusdescends toward the target landing area, and can update a depth map toinclude depth information of individual pixels of the new images of thelanding surface, wherein the individual pixels correspond to portions ofthe landing surface. The instructions can determine, based on the depthmap, whether the target landing area continues to satisfy a landingcondition as the apparatus descends.

In still further embodiments, an unmanned aerial vehicle (UAV) apparatusincludes a plurality of motors, a plurality of propellers, withindividual propellers carried by corresponding individual motors, atleast one camera, and a controller. The controller can be programmedwith instructions that, when executed, identify a target landing area ona landing surface based on at least one image of the landing surfacecaptured by the at least one camera, and direct the UAV apparatus toland using the motors at the target landing area. In particularembodiments, the at least one camera can include two bottom-viewingcameras, and the instructions, when executed, can simultaneously capturetwo images of the landing surface using the two bottom-viewing cameras.In any of the foregoing embodiments, the instructions, when executed,can capture two images of the landing surface using the at least onecamera at two different time points, wherein the two images of thelanding surface are captured from different perspectives. In any of theforegoing embodiments, the instructions, when executed, can direct theUAV apparatus to a predetermined height suitable for automatic terrainmeasurement. In any of the foregoing embodiments, the instructions, whenexecuted, can determine whether the at least one image of the landingsurface is captured when the UAV apparatus is directly above a center ofthe landing surface, and if the at least one image of the landingsurface is captured when the UAV apparatus is not directly above acenter of the landing surface, apply a perspective projection to the atleast one image of the landing surface. In any of the foregoingembodiments, the instructions, when executed, can direct the movableobject to move to a location directly above the target landing area.

In yet still further embodiments, a non-transitory machine readablemedium has stored thereon instructions that, when executed, receive arequest to land the movable objects, and in response to the request,identify a target landing area on a landing surface based on at leastone image of the landing surface obtained by the movable object. Theinstructions can direct the movable object to land at the target landingarea. In particular embodiments, the instructions, when executed, canmonitor an elevation of the movable object when the movable objectdescends toward the target landing area, and, if the elevation of themovable object is disruptively changing, stabilize a flight attitude ofthe movable object before continuing to descend. In any of the foregoingembodiments, the instructions, when executed, can detect a disruptivechange of the status of an inertial navigation system or a front-viewingcamera of the movable object, and pause the descent of the movableobject until the elevation of the movable object is no longerdisruptively changing. In any of the foregoing embodiments, theinstructions, when executed, can monitor an elevation of the movableobject when the movable object descends toward the target landing area,and, if the elevation of the movable object is disruptively changing,determine that the target landing area is currently not suitable forlanding. The moveable object can then be directed to hover at a currentaltitude. In any of the foregoing embodiments, the instructions, whenexecuted, can direct the movable object to ascend to a higher altitudeif the elevation of the movable object continues to be disruptivelychanging for a time period.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a partially schematic, isometric illustration of a UAV havinga control system configured to automatically evaluate a landing surface,in accordance with a representative embodiment of the presenttechnology.

FIG. 2 is a partially schematic illustration of a portion of the bottomof a UAV configured in accordance with embodiments of the presenttechnology.

FIG. 3 is a partially schematic illustration of a control systemincluding a first controller carried on board a UAV and configured tocontrol the UAV in accordance with representative embodiments of thepresent technology.

FIG. 4A is a flow diagram illustrating a planning stage of a process forlanding a movable object in accordance with representative embodimentsof the present technology.

FIG. 4B is a flow diagram illustrating a descending stage of a processfor landing a movable object in accordance with representativeembodiments of the present technology.

FIG. 5 is a partially schematic illustration of a sliding window and acorresponding candidate landing spot in accordance with a representativeembodiment of the present technology.

FIG. 6 is a flow diagram illustrating a process for identifying a flatarea in a pixel frame that corresponds to a target landing spot, inaccordance with a representative embodiment of the present technology.

FIG. 7 is a partially schematic illustration of a sample Gaussian blurfilter for candidate areas, in accordance with a representativeembodiment of the present technology.

FIG. 8 is a partially schematic illustration of a sample perspectiveprojection, in accordance with a representative embodiment of thepresent technology.

FIG. 9 is a partially schematic illustration of a hardware architectureof a computing device that performs the above processes, in accordancewith representative embodiments of the present technology.

DETAILED DESCRIPTION 1. Overview

Landing of UAVs is challenging. For example, a human operator canremotely control the landing process of a UAV. The human operatorcontrols the UAV to maneuver to a target location, e.g., in proximity tothe human operator. The UAV captures a live-view video stream of theground and transmits the live-view video stream in a real time to adevice viewed by the human operator. Based on the live-view of theground, the human operator identifies a suitable landing spot (alsoreferred to as target landing area) and manually controls the UAV toland on the landing spot.

However, such a landing process requires that the human operator behighly skilled in controlling the UAV. It is challenging to control theUAV landing process based on the live-view video feed from the UAV,particularly when the human operator does not have a direct visualperception of the UAV's flight trajectory and attitude. Thus, the UAVmay be damaged during a rough landing caused by human error during thelanding process.

The present technology is directed generally to devices thatautomatically measure or otherwise evaluate the terrain of a landingsurface or other target surface, and associated systems and methods.Such devices can include, e.g., UAVs, manned aircraft, autonomous cars,self-balancing vehicles, robots, etc. In particular embodiments, a UAVcan automatically evaluate a landing surface based on computervisualization technologies. During a landing process, the UAV capturesvisual signals (e.g., images or videos) of the landing surface. Based onthe visual signals, the UAV detects characteristics of the terrain ofthe landing surface and identifies a spot or area with a flat or smoothsurface suitable for landing. The UAV then descends onto the landingspot for an improved landing process.

The implementation of the technology can include on-board visual sensorssuch as cameras to evaluate the landing surface, e.g., based on stereomatching techniques. In particular embodiments, depth information isextracted from the images and used to reconstruct the 3D terrain andselect the target landing spot. Compared to conventional UAV landingprocesses, an advantage of this automatic landing surface measurementtechnology is that the UAV (or other types of movable objects) canautomatically and efficiently identify a target landing spot withoutintervention by a human operator. This in turn can allow the UAV todescend to the landing spot without damaging itself due to an uneven ortilted landing surface.

Several details describing structures or processes that are well-knownand often associated with UAVs (or other types of movable objects) andcorresponding systems and subsystems, but that may unnecessarily obscuresome significant aspects of the disclosed technology, are not set forthin the following description for purposes of clarity. Moreover, althoughthe following disclosure sets forth several embodiments of differentaspects of the technology, several other embodiments can have differentconfigurations or different components than those described in thissection. Accordingly, the technology may have other embodiments withadditional elements and/or without several of the elements describedbelow with reference to FIGS. 1-9.

FIGS. 1-9 are provided to illustrate representative embodiments of thedisclosed technology. Unless provided for otherwise, the drawings arenot intended to limit the scope of the claims in the presentapplication.

Many embodiments of the technology described below may take the form ofcomputer- or controller-executable instructions, including routinesexecuted by a programmable computer or controller. Those skilled in therelevant art will appreciate that the technology can be practiced oncomputer or controller systems other than those shown and describedbelow. The technology can be embodied in a special-purpose computer ordata processor that is specifically programmed, configured orconstructed to perform one or more of the computer-executableinstructions described below. Accordingly, the terms “computer” and“controller” as generally used herein refer to any data processor andcan include Internet appliances and handheld devices (including palm-topcomputers, wearable computers, cellular or mobile phones,multi-processor systems, processor-based or programmable consumerelectronics, network computers, mini computers and the like).Information handled by these computers and controllers can be presentedat any suitable display medium, including a CRT (cathode ray tube)display or LCD (liquid crystal display). Instructions for performingcomputer- or controller-executable tasks can be stored in or on anysuitable computer-readable medium, including hardware, firmware or acombination of hardware and firmware. Instructions can be contained inany suitable memory device, including, for example, a flash drive, USB(universal serial bus) device, and/or other suitable medium.

2. Representative Embodiments

FIG. 1 is a partially schematic, isometric illustration of arepresentative UAV 100 configured in accordance with embodiments of thepresent technology. The UAV 100 can include an airframe 110 that can inturn include a central portion 111 and one or more outer portions 112.In a representative embodiment shown in FIG. 1, the airframe 110includes four outer portions 112 (e.g., arms 113) that are spaced apartfrom each other as they extend away from the central portion 111. Inother embodiments, the airframe 110 can include other numbers of outerportions 112. In any of these embodiments, individual outer portions 112can support components of a propulsion system 120 that drives the UAV100. For example, individual arms 113 can support correspondingindividual motors 121 that drive corresponding propellers 122. The UAVcan control each individual set of a motor 121 and a correspondingpropeller 122, independent of the remaining motors 121 and propellers122.

The airframe 110 can carry a payload 130, for example, an imaging device131. In particular embodiments, the imaging device 131 can include acamera, for example, a camera configured to capture video data, stilldata, or both. The camera can be sensitive to wavelengths in any of avariety of suitable wavelength bands, including visual, ultraviolet,infrared or combinations thereof. In still further embodiments, thepayload 130 can include other types of sensors, other types of cargo(e.g., packages or other deliverables), or both. In many of theseembodiments, the payload 130 is supported relative to the airframe 110with a gimbal 150 that allows the payload 130 to be independentlypositioned relative to the airframe 110. Accordingly, for example whenthe payload 130 includes the imaging device 131, the imaging device 131can be moved relative to the airframe 110 to track a target. When theUAV 100 is not in flight, landing gear 114 can support the UAV 100 in aposition that protects the payload 130, as shown in FIG. 1.

In a representative embodiment, the UAV 100 includes a control system140 having some components carried on board the UAV 100 and somecomponents positioned off the UAV 100. For example, the control system140 can include a first controller 141 carried by the UAV 100, and asecond controller 142 (e.g., a human-operated, ground-based controller)positioned remote from the UAV 100 and connected to the first controller141 via a communication link 160 (e.g., a wireless link). The firstcontroller 141 can include an on-board computer-readable medium 143 athat executes instructions directing the actions of the UAV 100,including, but not limited to, operation of the propulsion system 120and the imaging device 131. The second controller 142 can include anoff-board computer-readable medium 143 b, and one or more input/outputdevices 148, e.g., a display 144 and control devices 145. The operatormanipulates the control devices 145 to control the UAV 100 remotely, andreceives feedback from the UAV 100 via the display 144 and/or otherdevices. In other representative embodiments, the UAV 100 can operateautonomously, in which case the second controller 142 can be eliminated,or can be used solely for operator override functions. The on-boardcomputer-readable medium 143 a can be removable from the UAV 100. Theoff-board computer-readable medium 143 b can be removable from thesecond controller 142, e.g., separable from the one or more input/outputdevices 148.

In a representative embodiment illustrated in FIG. 1, the UAV 100includes two front-viewing cameras 171 and 172. The cameras 171 and 172capture images or videos at a human-visible spectrum range, an infraredrange, an ultraviolet range, or a combination thereof. The UAV 100 caninclude more sensors, e.g., at the bottom. FIG. 2 is a partiallyschematic illustration of a portion of the bottom of a representativeUAV 100 configured in accordance with embodiments of the presenttechnology. At the bottom of the airframe 110, the UAV can furtherinclude two bottom-viewing cameras 173 and 174. In addition, the UAV caninclude two ultrasonic sensors 177 and 178. The ultrasonic sensors 177and 178 can detect objects or land surfaces below the UAV 100 andmeasure the distance to the objects or land surfaces by transmitting andreceiving ultrasonic waves. Although the UAV illustrated in FIGS. 1 and2 includes four cameras 171-174 and two ultrasonic sensors 177-178, UAVsin other embodiments can include any suitable number of cameras orultrasonic sensors. In still further embodiments, the UAV 100 caninclude other types of sensors.

FIG. 3 is a schematic illustration of the first controller 141, whichcan include a processor 146, memory 147, and input/output devices 148. Acontrol unit 149 (also referred to as controller) directs the operationof the various components of the UAV 100, including the propulsionsystem 120 described above. A computer-readable medium 143 (which can behoused in, or include elements of, any of the foregoing components)contains instructions that, when executed, direct the behavior of thepropulsion system 120 and other UAV systems. A first communicationdevice 161 a is configured to provide wireless communication with acorresponding second communication device 161 b carried by the secondcontroller 142, via the communication link 160.

Using the bottom-viewing cameras 173-174 or the ultrasonic sensors177-178, the UAV 100 is able to determine its flight height in relationto a ground level (e.g., a local round level), as well as evaluate theterrain of one or more candidate landing surfaces. Based on theevaluation, the UAV identifies a landing spot that is suitable (e.g.,generally flat) for landing within the candidate landing surface.

FIGS. 4A and 4B are flow diagrams illustrating a process 400 for landinga movable object in accordance with representative embodiments of thepresent technology. FIG. 4A illustrates a planning stage of the process400, and FIG. 4B illustrates a descending stage of the process 400. Theprocess 400 can be executed by a movable object of various types, suchas UAVs, manned aircraft, autonomous cars, self-balancing vehicles, orrobots.

Turning to FIG. 4A, at block 405 of process 400, the movable object(e.g., a UAV) receives an instruction to land at a target location. Atblock 410, the movable object navigates near the target location. Thereceiving of the instruction can be triggered by various mechanisms. Insome embodiments, for example, a human operator presses a “Go Home”button of the second controller 142 or a “Go Home” button on the display144 of the second controller 142 (as illustrated in FIG. 3). The secondcontroller 142 generates and transmits the instruction to the UAV inflight. The instruction being transmitted can include the coordinates ofthe target location. For example, the second controller 142 can includethe coordinates of the location of the second controller 142 itself asthe target location. Alternatively, the second controller 142 caninclude in the instruction a predetermined home location that isrecorded in a memory of the second controller 142.

In some embodiments, the instruction does not necessarily include thecoordinates of the target location. For example, when the UAV startsoperating and takes off, the UAV can detect and record the coordinatesof the takeoff location. Once the UAV receives the instruction, the UAVcan use the coordinates of the takeoff location as the recorded targetlocation for navigation.

In some embodiments, the UAV itself can generate the instruction whenthe battery level of the UAV is below a threshold level. The UAVgenerates the instruction for landing so that the UAV can avoid losingpower and crashing by returning to a landing spot before the battery isdepleted.

At block 415, the movable object descends to a predetermined heightsuitable for automatic terrain evaluation or measurement. The height(also referred to as elevation) is calculated in relation to an object(e.g., a building, a boat, or a platform) or ground below the UAV. Insome embodiments, the predetermined height is decided based on the pixelresolution of the bottom-viewing cameras 173 and 174 (as illustrated inFIG. 2). A three-dimensional (3D) reconstruction of the terrain can bebased on images captured by the bottom-viewing cameras 173 and 174. Anindividual pixel of the images represents a land area. The higher theUAV hovers, the larger the land area that is represented by theindividual pixel, and therefore the lower the resolution the 3D terrainreconstruction. The predetermined height is generally small enough suchthat the resolution of the 3D terrain reconstruction is high enough toidentify a suitable landing spot for the UAV.

On the other hand, the predetermined height is large enough such thatthe field of view of the bottom-viewing cameras 173 and 174 covers asuitable large area of the land. If the predetermined height is toosmall, the cameras 173-174 can only cover a small land area, which maynot include any suitable landing spots. In some embodiments, thepredetermined height is about 3 meters.

The UAV can use various types of depth sensors to determine the currentflight height. For example, the depth sensors for determining flightheight can include a single camera, a pair of cameras, an ultrasonicsensor, an IMU, or a laser rangefinder. In some embodiments, the UAV canuse the bottom-viewing cameras 173 and 174 (as illustrated in FIG. 2) todetermine the flight height. As illustrated in FIG. 2, thebottom-viewing cameras 173 and 174 are disposed on the bottom of the UAV100 and are separated by a known distance. In other words, thebottom-viewing cameras 173 and 174 capture images or videos of theobject or ground from different perspectives. The UAV 100 can use stereomatching techniques (e.g., semi-global block matching) to findcorresponding pixels in the images or video frames captured by cameras172 and 174, and conduct a 3D reconstruction process by triangulation.In other embodiments, the cameras 172 and 174 can be replaced by one ormore external cameras (e.g., the imaging device 131 illustrated in FIG.1).

In still another embodiment, the UAV can use a single bottom-viewingcamera to capture images or videos of the objector ground from differentperspectives. For example, the camera of the UAV captures a first imageof the ground from a first perspective. Then the UVA moves to adifferent location, and the camera captures a second image of the groundfrom a second perspective. The UAV can then use stereo matchingtechniques to find corresponding pixels in those two images and conduct3D reconstruction by triangulation.

In yet another embodiment, the UAV can use one or more ultrasonicsensors (e.g., ultrasonic sensors 177 and 178 as illustrated in FIG. 2)to determine the flight height. An ultrasonic sensor of the UAV emits anultrasonic wave downward to the object or ground. The emitted ultrasonicwave is reflected by the object or ground. The ultrasonic sensorreceives the reflected ultrasonic wave and determines the height basedon the travelling time of the ultrasonic wave.

Furthermore, the UAV 100 can use an on-board inertial measurement unit(IMU) to determine the flight height. The IMU can include, e.g., one ormore accelerometers, gyroscopes, magnetometers, or a combinationthereof. The IMU provides acceleration information, which can be used tocalculate velocity and position information by integral operations.

In some embodiments, the UAV 100 can use a combination of multiplecomponents, such as the ultrasonic sensors 177 and 178, thebottom-viewing cameras 173 and 174 and the IMU, to determine the flightheight. In some other embodiments, the UAV 100 dynamically switchesbetween components, such as the ultrasonic sensors 177 and 178, thebottom-viewing cameras 173 and 174 and the IMU, to determine the flightheight.

At block 416, the movable object captures images using one or morecameras (e.g., the cameras 172 and 174 of the UAV 100 as illustrated inFIG. 1). At decision block 417, the movable object determines whetherthe images are captured when the UAV is at a horizontal state. In otherwords, the movable object determines whether the images are captured forthe landing surface directly below the UAV. If not, at block 418, themovable object applies a perspective projection, which is describedlater with reference to FIG. 8. Otherwise, the process proceeds directlyto block 420.

At block 420, the movable object determines a size of a sliding windowin the pixel frame of the cameras 172 and 174 that corresponds a size ofa target landing spot in the ground frame. FIG. 5 is a partiallyschematic illustration of a sliding window and a corresponding suitabletarget landing spot, in accordance with a representative embodiment ofthe present technology. A target landing spot 515 is located in a groundframe 510. The target landing spot 515 has a radius R. A representativecircular sliding window 525 is located in a pixel frame 520. The slidingwindow 525 has a radius r. Although FIG. 5 show the target landing spot515 and the sliding window 525 having circular shapes, the targetlanding spot 515 and the sliding window 525 can have other suitableshapes, such as squares, rectangles, etc.

The pixel frame 520 represents an area of pixels in the bottom-viewingcameras 173 and 174 that captures images or videos of the ground frame510 on the ground. In other words, an individual pixel of the pixelframe 520 contains visual information of a corresponding area within theground frame 510 on the ground. The target landing spot 515 in theground frame 510 has a size that is large enough for the UAV to land.Thus, the size of the target landing spot 515 can depend on the size(e.g., the “footprint” or planform size) of the UAV. The planform is thecontour of the UAV (or other types of movable objects) or mass as viewedfrom above. In some embodiments, the target landing spot 515 has a sizethat equals the size of the UAV. Similarly, the shape of the targetlanding spot 515 can also relate to the shape of the UAV. In someembodiments, for example, the shape of the target landing spot 515 cangenerally conform to the shape (e.g., the planform shape) of the UAV.

The sliding window 525 represents an area in the pixel frame 520 thatcaptures the image of the target landing spot 515. As shown in FIG. 5,the radius r of the sliding window 525 depends on the size of the UAV,the current height, and the focal lengths of the cameras 172 and 174:r=f*R/H, where R is the radius of the target landing spot, f is thefocal length, and H is the focal length plus the current flight heightmeasured between the camera and the land surface.

Turning back to FIG. 4A, at block 425, the movable object scans thepixel frame using the sliding window to identify a flat area thatcorresponds to a target landing spot. The details of the process carriedout in block 425 are described later with reference to FIG. 6.

Then at decision block 430, the movable object determines whether a flatarea is identified. If the UAV identifies no flat area within the pixelframe at decision block 430, block 485 includes navigating around thetarget location. The process 400 further goes back to block 416 tocapture images of another area for identifying potential landing spots.

If the UAV identifies a flat area that corresponds to a target landingspot, then the process 400 proceeds to the descending stage, which isillustrated in FIG. 4B. Turning to FIG. 4B, at block 435, the movableobject navigates and descends to the identified flat area as the targetlanding spot. Furthermore, at block 440, the movable object monitorselevation in real time during the descent process. At decision block445, the movable object determines whether there is a disruptive changein flight elevation. The disruptive change in flight elevation can bedetected by the bottom-viewing cameras 173 and 174, the IMU, or acombination thereof. Since the UAV descends gradually, such a disruptivechange may suggest a change of the environment, particularly a change ofthe landing surface. For example, an obstruction may have moved onto thelanding surface.

In some embodiments, the movable object determines that a change isdisruptive if the change in flight elevation is more than a predictedchange in flight elevation based on the current descending speed of themoveable object (e.g., clue to a downdraft). In some other embodiments,the movable object determines that a change is disruptive if the changein flight elevation is faster than a maximum descending speed at whichthe moveable object is designed to operate.

If there is no disruptive change in flight elevation, at block 450, themovable object continues descending until the UAV lands. If there is adisruptive change in flight elevation, at block 455, the movable objectchecks the status of another sensor, for example, an inertial navigationsystem or front-viewing cameras (e.g., front-viewing cameras 171 and 172illustrated in FIG. 1) of the UAV. Decision block 460 includesdetermining if there is a further disruptive change in the status of theinertial navigation system or front-viewing cameras. If there is afurther disruptive change in the status of the inertial navigationsystem or front-viewing cameras, this generally indicates that theflight altitude of the UAV is unstable. Thus, at block 465, the movableobject stabilizes UAV flight altitude before continuing to descend.

If there is a no disruptive change in the status of the inertialnavigation system or front-viewing cameras, this generally indicatesthat the UAV itself is descending in a stable matter. Instead, thedisruptive change in flight elevation is caused by an environment change(e.g., a person or an object is now on the landing spot). At block 470,the movable object pauses the descending process and hovers. At decisionblock 475, the machine determines whether the flight elevation readingreturns to a normal level within a certain (e.g., pre-selected) timeperiod. If so, at block 480, the movable object continues descendinguntil the UAV lands. Otherwise, the process 400 goes back to block 416of the planning stage (illustrated in FIG. 4A) to repeat the process ofidentifying another target landing spot.

In some embodiments, a controller (e.g., the first controller 141 asillustrated in FIGS. 1 and 3) can be programmed with instructions, whenexecuted, conduct one or more steps as illustrated in FIGS. 4A and 4B.For example, an apparatus can include a structure and one or more motorscarried by the structure. The structure can include, e.g., at least aportion of a UAV, a manned aircraft, an autonomous car, a self-balancingvehicle, or a robot. The apparatus can further include such a controllerprogrammed to conduct one or more steps of the process 400.

FIG. 6 is a flow diagram illustrating a process 600 for identifying aflat area in a pixel frame that corresponds to a target landing spot, inaccordance with a representative embodiment of the present technology.At block 605, the movable object (e.g., UAV) identifies a candidate areawithin the pixel frame of the images by moving a sliding window to alocation within the pixel frame. The size of the sliding window isdetermined, e.g., based on the technology described above with referenceto FIG. 5. The individual candidate area occupies a unique portion ofthe pixel frame. The candidate areas can, but need not necessarily,overlap with each other. In some embodiments, the technology candetermine a step size (measured by, e.g., pixels) for moving the slidingwindow. For example, once a candidate area is identified, the nextcandidate area can be determined by moving the sliding window from thecurrent candidate area by the determined step size.

At block 610, the movable object retrieves the location coordinates anddepth coordinate of an individual pixel within the candidate area. Forexample, an individual pixel can have three coordinates. The x and ycoordinates denote the location of the pixel within the pixel frame. Thez coordinate denotes the depth information of the portion of the landingsurface represented by the pixel. Thus, the x, y, z coordinates of thepixels recreate the terrain of a landing area represented by thecandidate area. In some embodiments, the depth information can beretrieved using stereo matching techniques (e.g., semi-global blockmatching) based on images simultaneously captured by two cameras (e.g.,cameras 172 and 174 as illustrated in FIG. 1) or two sequential imagescaptured by a single camera. In other embodiments, the depth informationcan be retrieved using another distance sensor such as an ultrasonicsensor.

At block 615, the movable object identifies a “best fit” plane fit tothe coordinates of the pixels of the candidate area, and a cost functionassociated with the best fit plane, which denotes a degree of deviationof pixels from the plane. The best fit plane is a plane with theminimized cost values among the possible fit planes. In other words, thedegree of deviation suggests whether the pixels of the candidate areacan be treated as a flat plane. If the degree of deviation is small, theactual landing surface represented by the candidate area is consideredto be flat and suitable for a proper landing.

In some embodiments, the movable object can use an algorithm, e.g.,Leverberg-Marquardt algorithm, to find the best fit plane and itsassociated cost value. The algorithm minimizes a sum of squares ofdeviations of pixels Y_(i) from the fit plane function f.

S(β)=Σ_(i=1) ^(n)[Y _(i)−ƒ(P _(w,i),β)]².

The fit plane function can be expressed as:

${f\left( {P_{w,i},\beta} \right)} = {\frac{{{ax} + {by} + {cz} + d}}{\sqrt{a^{2} + b^{2} + c^{2}}} + {ɛ{{{a^{2} + b^{2} + c^{2} + d^{2} + 1}}.}}}$

The residual vector r is:

$r = {{Y_{i} - {f\left( {P_{w,i},\beta} \right)}} = {{- \frac{{{ax} + {by} + {cz} + d}}{\sqrt{a^{2} + b^{2} + c^{2}}}} - {ɛ{{{a^{2} + b^{2} + c^{2} + d^{2} + 1}}.}}}}$

The cost function for calculating the cost value is:

min C=Σ _(i=1) ^(n) r ².

The residual can be further damped, e.g., according to the followingalgorithm:

(J ^(T) J+λI)δ=J ^(T) r.

where J is the Jacobian matrix of derivatives of the residuals withrespect to the parameters, λ is a damping parameter (adaptive balancebetween the 2 steps), and r is the residual vector.

The dampening process first initializes values for damping parameter,and evaluates the residuals r and the Jacobian J at the initialparameter guess. Then the dampening process calculates the metric,g=J^(T)J+λI, and the cost gradient, ∇C=J^(T)r, C=½ r².

Then the dampening process evaluates the new residuals r_(new) at thepoint given by x_(new)=x−g⁻¹∇C, and calculates the cost at the new pointC_(new)=½r_(new) ². If C_(new)<C, the process accepts the step,x=x_(new), and set r=r_(new) and λ=λ/λ_(down). Otherwise, the processrejects the step, and keeps the old parameter guess x and the oldresiduals r, and adjusts λ=λ/λ_(up). The dampening process checks forconvergence. If there is a convergence, the process returns x as thebest-fit parameter. If there is no convergence yet but the step isaccepted, the process evaluates the Jacobian J at the new parametervalues and goes back to calculate the metric g and the cost gradient.

At decision block 620, the movable object determines whether allcandidate areas within the pixel frame have been processed. If not, theprocess 600 goes back to block 605 to identify another candidate areawithin the pixel frame. If all candidate areas within the pixel framehave been processed, at block 625, the movable object discards candidateareas with cost values that are larger than a threshold value. Thediscarded areas are considered not flat because the associated costvalues are large, which means their pixels are not close to the best fitplanes. In other words, the corresponding landing surface portion is notflat, or not sufficiently flat.

At block 630, the movable object further discards candidate areas thatare tilted. The tilted candidate areas represent tilted landingsurfaces, which are not suitable for a proper landing. In someembodiments, for an individual candidate area, the movable object cancompare the direction of a gravity vector (which can determined by,e.g., the IMU) with the direction of a normal vector of the best fitplane of the candidate area. If the angle between the directions of thegravity vector and the normal vector exceeds a threshold value, thecorresponding landing surface is too tilted to be used for landing.

At block 640, the movable object can apply a smoothing function toadjust the cost functions of individual remaining candidate areas suchthat the adjusted cost functions include information of cost functionsof nearby candidate areas. The reason for the smoothing function isthat, in at least some embodiments, the goal is not just to pick acandidate area with the least cost function (which means a flatsurface), but to pick a candidate area with nearby areas that are alsorelatively flat. This is beneficial because the UAV or other device maydeviate from the planned path during the landing, and may land on anearby area instead of the picked candidate area. Therefore, picking acandidate area with nearby areas that are also relatively flat reducesthe chances for damaging the UAV during landing.

In some embodiments, the smoothing function can be, e.g., a Gaussianblur function. For example, FIG. 7 is a partially schematic illustrationof a Gaussian blur filter, in accordance with a representativeembodiment of the present technology. The 3 by 3 Gaussian blur filterillustrated in FIG. 7 can be applied to, e.g., block 640 of process 600.As shown in FIG. 7, the Gaussian blur filter takes ¼ of the originalcost function of the candidate area, and ⅛ of cost functions of left,right, upper and lower neighboring areas, as well as 1/16 of costfunctions of cornering areas.

Returning to FIG. 6, at block 650, the movable object selects acandidate area with the least cost value calculated from the adjustedcost function as a flat area that corresponds to a target landing spot.The cost value of the adjusted cost function, adjusted by the smoothingfunction (e.g., a Gaussian blur filter), not only indicates whether thecandidate area is flat, but also indicates a degree of flatness fornearby areas. The UAV can then descend to the identified target landingspot according to the process illustrated in FIGS. 4A and 4B.

As the process 400 in FIGS. 4A and 4B illustrates, the movable objectcan apply a perspective projection to the images if the images not arecaptured when the UAV is at a horizontal state. FIG. 8 is a partiallyschematic illustration of a perspective projection, in accordance with arepresentative embodiment of the present technology. As shown in FIG. 8,a pixel frame 820 of the captured images does not correspond to alanding surface directly below the UAV. The area 825 within the pixelframe 820 has a corresponding perspective projected area 815 on theground frame 810. In various embodiments, the area 825 can havedifferent shapes, such as circles, ellipses, squares, rectangles, etc.Accordingly, the projected area 815 can have shapes such as circles,ellipses, squares, rectangles, etc. For example, the area 825 can becircular and the projected area 815 can be elliptical.

Assuming [u,v] are coordinates of a pixel within the area 825 within thepixel frame 820, the coordinates of the corresponding point on theperspective projected area 815 are denoted as [x_(w), y_(w), z_(w)]. Theperspective projection operation is:

$\begin{bmatrix}u \\v \\1\end{bmatrix} = {{{K\lbrack{RT}\rbrack}\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix}}.}$

K is the intrinsic matrix, which includes parameters associated with thecamera, such as focal length, image sensor format, principal point ofthe camera lens and lens distortion. R and T include extrinsicparameters, which denote the coordinate system transformation from worldcoordinates [x_(w), y_(w), z_(w)] to camera coordinates [u,v]. R is arotation matrix. T represents the position of the origin of the worldcoordinate system expressed in coordinates of the camera-centeredcoordinate system. If the coordinate system is chosen such that theorigin is at the center of the projected area, then z_(w)=0, x_(w)²+y_(w) ²=1.

In other words, y_(w)=√{square root over (1−x_(w) ²)}. Thus,

$\begin{bmatrix}u \\v \\1\end{bmatrix} = {{{K\lbrack{RT}\rbrack}\begin{bmatrix}x_{w} \\\sqrt{1 - x_{w}^{2}} \\0 \\1\end{bmatrix}}.}$

FIG. 9 is a partially schematic illustration of a hardware architectureof a computing device that can perform the above processes, inaccordance with representative embodiments of the present technology.The computing device 900 can execute some or all of the processorexecutable process steps that are described below in detail. In variousembodiments, the computing device 900 includes a processor subsystemthat includes one or more processors 902. Processor 902 may be or mayinclude, one or more programmable general-purpose or special-purposemicroprocessors, digital signal processors (DSPs), programmablecontrollers, application specific integrated circuits (ASICs),programmable logic devices (PLDs), or the like, or a combination of suchhardware based devices.

The computing device 900 can further include a memory 904, a networkadapter 910, a cluster access adapter 912 and a storage adapter 914, allinterconnected by an interconnect 908. Interconnect 908 may include, forexample, a system bus, a Peripheral Component Interconnect (PCI) bus, aHyperTransport or industry standard architecture (ISA) bus, a smallcomputer system interface (SCSI) bus, a universal serial bus (USB), oran Institute of Electrical and Electronics Engineers (I9E) standard 1394bus (sometimes referred to as “Firewire”) or any other datacommunication system.

The cluster access adapter 912 includes one or more ports adapted tocouple the computing device 900 to other devices. In the illustratedembodiment, Ethernet can be used as the clustering protocol andinterconnect media, although other types of protocols and interconnectsmay be utilized within the cluster architecture described herein.

The computing device 900 can be embodied as a single- or multi-processorsystem executing an operating system 906 that can implement a high-levelmodule, e.g., a manager, to logically organize the information as ahierarchical structure of named directories, files and special types offiles called virtual disks at the storage devices. The computing device900 can further include graphical processing unit(s) for graphicalprocessing tasks or processing non-graphical tasks in parallel.

The memory 904 can comprise storage locations that are addressable bythe processor(s) 902 and adapters 910, 912, and 914 for storingprocessor executable code and data structures. The processor 902 andadapters 910, 912, and 914 may, in turn, comprise processing elementsand/or logic circuitry configured to execute the software code andmanipulate the data structures. The operating system 906, portions ofwhich are typically resident in memory and executed by the processor(s)902, functionally organizes the computing device 900 by (among otherthings) configuring the processor(s) 902 to invoke. It will be apparentto those skilled in the art that other processing and memoryimplementations, including various computer readable storage media, maybe used for storing and executing program instructions pertaining to thetechnology.

The network adapter 910 can include multiple ports to couple thecomputing device 900 to one or more clients over point-to-point links,wide area networks, virtual private networks implemented over a publicnetwork (e.g., the Internet) or a shared local area network. The networkadapter 910 thus can include the mechanical, electrical and signalingcircuitry needed to connect the computing device 900 to the network.Illustratively, the network can be embodied as an Ethernet network or aFibre Channel (FC) network. A client can communicate with the computingdevice over the network by exchanging discrete frames or packets of dataaccording to pre-defined protocols, e.g., TCP/IP.

The storage adapter 914 can cooperate with the operating system 906 toaccess information requested by a client. The information may be storedon any type of attached array of writable storage media, e.g., magneticdisk or tape, optical disk (e.g., CD-ROM or DVD), flash memory,solid-state disk (SSD), electronic random access memory (RAM),micro-electro mechanical and/or any other similar media adapted to storeinformation, including data and parity information. The storage adapter914 can include multiple ports having input/output (I/O) interfacecircuitry that couples to the disks over an I/O interconnectarrangement, e.g., a conventional high-performance, Fibre Channel (FC)link topology. In various embodiments, the cluster adapter 912 and thestorage adapter 914 can be implemented as one adapter configured toconnect to a switching fabric, e.g., a storage network switch, in orderto communicate with other devices and the mass storage devices.

One feature of several of the embodiments described above is that theUAV can automatically evaluate or measure a landing surface usingcomputer visualization technologies. The UAV can use its visual sensorssuch as cameras to measure the landing surface, e.g., using stereomatching techniques. The depth information can be extracted from theimages and used for reconstructing the 3D terrain and selecting thetarget landing spot. An advantage of this arrangement is that the UAVcan automatically and efficiently identify a target landing spot withoutintervention by a human operator. Thus, the UAV can properly descend tothe landing spot without damaging itself due to an uneven or tiltedlanding surface.

From the foregoing, it will be appreciated that specific embodiments ofthe technology have been described herein for purposes of illustration,but that various modifications may be made without deviating from thetechnology. For example, the terrain measurement technology can beapplied devices other than UAVs, e.g., manned aircraft, autonomous cars,self-balancing vehicles, robots, wearable devices such as smart glasses,virtual reality (VR) or augmented reality (AR) had-mounted display. Forinstance, an autonomous car or a robot can include one or more camerasfor capturing images of the land surface in front of the autonomous caror robot. The autonomous car or robot measures the terrain condition byextracting the depth information from the images. Based on the terraincondition, the autonomous car or robot automatically selects a routewith a smooth driving surface.

The landing site is described above in the content of severalembodiments as including the ground. In other embodiments, the landingsite can include objects (e.g., a boat, vehicle, or platform), water, orother suitable sites.

The present technology can also be directed generally to wearabledevices such as smart glasses, virtual reality (VR) or augmented reality(AR) head-mounted displays, etc. The technology can automaticallymeasure or otherwise evaluate the terrain of a surface in front of sucha wearable device or a user wearing the wearable device and identify atarget surface that is suitable for the user to travel. The technologycan further indicate the target surface to the user using the wearabledevice or other devices that are capable of outputting the informationof the target surface to the user.

Certain aspects of the technology described in the context of particularembodiments may be combined or eliminated in other embodiments. Forexample, in some embodiments, the UAV does not need to discard a tiltedcandidate area, if the UAV is capable of landing on a tilted surface. Insome embodiments, the UAV does not need to apply perspective projectionto captured images, if the UAV is equipped with a mechanism toautomatically orient the camera(s) to capture images downward,regardless of the flight attitude of the UAV. In other embodiments, theUAV does not apply a smoothing function to the cost functions of thecandidate areas, e.g., if the UAV is capable of descending down to alanding spot in an accurate manner.

Further, while advantages associated with certain embodiments of thetechnology have been described in the context of those embodiments,other embodiments may also exhibit such advantages, and not allembodiments need necessarily exhibit such advantages to fall with withinthe scope of the present technology. Accordingly, the present disclosureand associated technology can encompass other embodiments not expresslyshown or described herein.

To the extent any materials incorporated herein conflict with thepresent disclosure, the present disclosure controls.

At least a portion of the disclosure of this patent document containsmaterial which is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument or the patent disclosure, as it appears in the Patent andTrademark Office patent file or records, but otherwise reserves allcopyright rights whatsoever.

1.-30. (canceled)
 31. A computer-implemented method for controllinglanding of a movable object, comprising: receiving a request to land themovable object on a surface; in response to the request, obtaining adepth map of the surface, each pixel of the depth map corresponding to aportion of the surface; controlling the movable object to land based onflatness or tilt of the depth map.
 32. The computer-implemented methodof claim 31, wherein obtaining the depth map of the surface comprises:performing a stereo matching on images captured by two cameras at thesame time or two sequential images captured by a single camera.
 33. Thecomputer-implemented method of claim 31, wherein a distance sensor isused to obtain the depth map of the surface.
 34. Thecomputer-implemented method of claim 31, further comprising: determininga fitting plane based on position information and depth information ofeach pixel in the depth map; determining the flatness as a deviation ofthe depth map from the fitting plane; and determining the tilt of thedepth map based on an angle between a normal vector normal to thefitting plane and a gravity vector extending in a direction of gravity.35. The computer-implemented method of claim 31, wherein controlling themovable object to land based on the flatness or tilt of the depth mapcomprises: determining whether the surface meets one or more landingconditions based on the flatness or tilt of the depth map; controllingthe movable object to stop landing when the one or more landingconditions are not met.
 36. The computer-implemented method of claim 35,wherein controlling the movable object to stop landing comprises:guiding the movable object to another location to identify a possiblelanding site.
 37. The computer-implemented method of claim 35, whereincontrolling the movable object to land based on the flatness or tilt ofthe depth map further comprises: controlling the movable object tocontinue landing when the one or more landing conditions are met. 38.The computer-implemented method of claim 31 further comprising:controlling the movable object to descend to a predetermined heightbefore obtaining the depth map.
 39. The computer-implemented method ofclaim 38, wherein the depth map is obtained based on an image capturedby a bottom-viewing camera when the movable object is at thepredetermined height.
 40. The computer-implemented method of claim 39,wherein the predetermined height is determined based on a pixelresolution of the bottom-viewing camera.
 41. An unmanned aerial vehicle(UAV) apparatus, comprising: a plurality of motors; a plurality ofpropellers; a controller programmed with instructions that, whenexecuted: receive a request to land the UAV apparatus on a surface; inresponse to the request, obtain a depth map of the surface, each pixelof the depth map corresponding to a portion of the surface; control theUAV apparatus to land based on flatness or tilt of the depth map. 42.The unmanned aerial vehicle apparatus of claim 41, wherein obtaining thedepth map of the surface comprises: performing a stereo matching onimages captured by two cameras at the same time or two sequential imagescaptured by a single camera.
 43. The unmanned aerial vehicle apparatusof claim 41, wherein a distance sensor is used to obtain the depth mapof the surface.
 44. The unmanned aerial vehicle apparatus of claim 41,wherein the instructions, when executed: determine a fitting plane basedon position information and depth information of each pixel in the depthmap; determine the flatness as a deviation of the depth map from thefitting plane; and determine the tilt of the depth map based on an anglebetween a normal vector normal to the fitting plane and a gravity vectorextending in a direction of gravity.
 45. The unmanned aerial vehicleapparatus of claim 41, wherein controlling the UAV apparatus to landbased on the flatness or tilt of the depth map comprises: determiningwhether the surface meets one or more landing conditions based on theflatness or tilt of the depth map; controlling the UAV apparatus to stoplanding when the one or more landing conditions are not met.
 46. Theunmanned aerial vehicle apparatus of claim 45, wherein controlling theUAV apparatus to stop landing comprises: guiding the UAV apparatus toanother location to identify a possible landing site.
 47. The unmannedaerial vehicle apparatus of claim 45, wherein controlling the UAVapparatus to land based on the flatness or tilt of the depth map furthercomprises: controlling the UAV apparatus to continue landing when theone or more landing conditions are met.
 48. The unmanned aerial vehicleapparatus of claim 41, wherein the instructions, when executed: controlthe UAV apparatus to descend to a predetermined height before obtainingthe depth map.
 49. The unmanned aerial vehicle apparatus of claim 48,wherein the depth map is obtained based on an image captured by abottom-viewing camera when the UAV apparatus is at the predeterminedheight, wherein the predetermined height is determined based on a pixelresolution of the bottom-viewing camera.
 50. A non-transitorycomputer-readable medium storing instructions that, when executed, causeone or more processors to perform a method for controlling landing of amovable object, the method comprising: receiving a request to land themovable object on a surface; in response to the request, obtaining adepth map of the surface, each pixel of the depth map corresponding to aportion of the surface; controlling the movable object to land based onflatness or tilt of the depth map.